Engaging Students through Video: Integrating Assessment and Instrumentation

CS50 is Harvard’s introductory course for majors and non-majors alike. For years, we have posted videos of the course’s lectures and sections online for the sake of review and distance education alike. But students’ experience with these videos has been historically passive. Students have been able to watch the course’s content on demand, rewinding and fast-forwarding at will, but they have not had means to engage interactively with the content or to check their understanding of material while watching videos. Furthermore, while we collected basic usage data (e.g., how many times a video was viewed), we lacked detailed analytics describing, for example, which portions of a video were commonly skipped or watched multiple times by students. To make videos more immersive and engaging for students, we developed CS50 Video, an open-source video player for desktop and mobile devices. CS50 Video allows instructors to integrate assessment questions to be answered by students at their own pace or at specific points in time directly into a video player. CS50 Video also allows students to search over video transcripts to find content easily as well as view videos at variable playback speeds (in order to make videos more accessible for ESL learners). Finally, CS50 Video integrates with third-party analytics solutions to allow instructors to view detailed usage statistics describing how students are interacting with videos (e.g., which videos or portions of videos are commonly watched or skipped over). We have deployed CS50 Video to students taking CS50 online and have obtained preliminary results. Because CS50 Video stores responses to questions server-side, we have been able to track students’ performance on in-video assessments. Thus far, we have observed that only 28% of students who watch online videos have engaged with assessment questions. Students who answer an assessment question incorrectly on their first attempt will often try again until reaching a correct answer, with 84.5% of correct answers reached in at most three attempts. We next plan to analyze the effects of in-video assessments on students’ mastery of material and introduce A/B-testing functionality for questions. We also plan to use students’ performance on assessments to understand the topics with which students struggle. INTRODUCTION CS50 is Harvard’s introduction to computer science for majors and non-majors alike, a one-semester course combining concepts typically taught in CS1 and CS2. Over the course of a 14-week semester, students are expected to attend two 90-minute lectures and one 90-minute section per week while completing eight problem sets, two exams, and a final project. Traditionally, all of CS50’s lectures and sections have been filmed, with videos posted online within 24 hours in downloadable and streaming formats. On-campus students take advantage of these videos for the sake of review (and in cases of missed classes), while distance learners rely on these videos in order to take the course from afar. Prior to Fall 2012, students watched videos via a traditional, web-based player consisting only of basic playback controls, including rewind and fast-forward. However, that interface offered students a passive experience. Not only were students unable to ask questions of the course’s staff as they watched videos, but instructors were unable to assess students’ understanding of material in situ. Similarly, because little was logged, instructors were unable to determine how students interacted with videos and were generally unaware which points in a video were commonly skipped, which videos were most utilized by students, and which videos were most confusing for students. Finally, students were not able to search over video content and easily find the points in a video at which a particular topic was discussed. And so, to create a more interactive experience for students with videos online, we developed CS50 Video, a video player that integrates assessments (interactive questions and answers) as well as full-text searchability, with support for desktop and mobile devices alike. With CS50 Video, instructors can embed a questions in a variety of formats within a video to be answered by students at their own pace. CS50 Video also allows instructors, via third-party analytics integration, to view detailed usage statistics describing which questions were particularly confusing for students and which portions of videos were commonly repeated or skipped over. Moreover, using CS50 Video’s mobile player, students are able to view overlays of various classroom materials while viewing a video, creating a more immersive experience. We have deployed CS50 Video to 86,400 online students and have begun to assess the results. Using CS50 Video, we have integrated questions into the course’s lecture and section videos, and we have begun to track students’ progress through CS50 Video’s server-side logging of responses. With these data, we have begun to investigate relationships among a number of variables and learning outcomes, among them: • Time of Question—Are more questions answered correctly if they are presented during a video or after a video? • Type of Question—Do students learn more effectively by answering multiple-choice or free-response questions? • Frequency of Questions—Does a higher number of assessment questions necessarily lead to higher levels of understanding? We present in this paper the pedagogy behind and design of CS50 Video, along with preliminary results. In Section 2, we present background for our experiment, and in Section 3, we explore related works. In Section 4, we present questions we seek to answer using data obtained from CS50 Video. In Section 4, we identify problems that we aim to solve using CS50 Video, and in Section 5, we present our preliminary findings. Finally, in Section 6, we describe future work and our roadmap for CS50 Video’s development. BACKGROUND Video players abound on the web, but none offer quite the same feature set as CS50 Video. YouTube [4], for example, allows users to enable captions, adjust the playback speed of videos, and seek through content, fast-forwarding and rewinding as necessary. Khan Academy [8] and edX [2] offer much of their course content through embedded YouTube players. Whereas YouTube allows users to view and navigate content, CS50 Video affords interaction, allowing users to view and answer in-lecture questions. Udacity [9] hosts its lecture videos on YouTube, but also offers one interactive question per video. At the end of a video clip, the question replaces the video content. Students are then able to submit an answer as many times as they'd like until they get the question right and then move on to the next video clip. Udacity also provides solution videos that walk students through the answer to each question. While Udacity offers one question per short clip, CS50 Video allows users to view multiple questions during longer video segments, allowing lecturers more flexibility in designing their video content. Coursera [1] offers an alternative model. Coursera does not host its videos on YouTube, and, as a result, has more flexibility in displaying their questions. Coursera provides longer videos that are broken up by interactive questions, much like Udacity. Unlike Udacity, which has 1–2 minute video clips, Coursera's lesson videos are 15–20 minutes in length and have 2–3 questions each. Like Udacity, Coursera questions can be submitted and checked against an answer key. Coursera’s offering is the closest in style to CS50 Video. Whereas Coursera’s lecture content is designed around the in-lecture questions, CS50 Video allows lecturers to add questions to already existing content, and allows users to open questions at their own discretion. CS50 Video alerts students when questions are available (based on their location in a particular video) via a flashing icon, thereby allowing students to engage interactively when ready. However, students’ experience is still guided by the instructor, as teaching staff can choose to allow students to explore questions non-linearly or automatically pause the video for assessments. RELATED WORK Current literature on the efficacy of online classes has focused on issues surrounding student engagement and on allowing students to ask questions of remote professors [6]. Much of the work in this space has focused on assessment following a standard format in which students could access unit quizzes after watching a given set of videos. In particular, Harwood and McMahon found that introducing videos into a curriculum increased student performance on comprehension assessments [5]. Foertsch et al., meanwhile, specifically evaluated the effectiveness of videos themselves in instruction [3]. Rather than holding traditional lectures, instructors engaged students in interactive problem sessions. For homework, students were required to watch pre-recorded lecture videos that introduced concepts and material. Students performed better (and rated the course more positively) when they could navigate and consume lecture content online and at their own pace. Our work differs in that we are attempting to determine as directly and specifically as possible which segments of video content are effective. Whereas a “unit quiz” can afford insights into the effectiveness of a set of lectures, mid-video questions can more directly ask “Did this part of lecture effectively cover the topic?” before the student has a chance to turn to other resources. Mavlankar et al. created ClassX, an automated lecture video capture application that allows students to view different parts of the classroom (e.g., the chalkboard or lecture slides) at their own choosing during a lecture video [7]. They found that students were, in fact, engaged in swapping perspectives and interested in viewing different parts of the classroom. This result influenced the design of CS50 Video’s interactive overlays that display different classroom resources on mobile devices. Zhang et al. investigated the claim that adding video content to a curriculum will reliably improve student performance [10]. While they found that adding video alone is not always sufficient, they concluded that interactive video content, in which students were in control of the content being displayed (as via a table of contents), did tend to lead to higher performance. PROBLEM The experiences of passively watching a video online and actively engaging in a lecture in person are markedly different. During an on-campus lecture, an instructor has the ability to pose questions to students and give individualized feedback based on responses. In an online environment, however, these questions are at best rhetorical, and students’ ability to receive feedback on their responses is limited. As a result, students are unable to check their understanding of a topic while watching a video. Similarly, non-classroom environments may prove to be more distracting for students, and the resulting potential decrease in engagement may be detrimental to learning outcomes. We considered introducing assessment questions into the online video experience by directing students to a list of questions before or after a video. For example, we could have broken up our 90-minute lectures into 5–10 minute segments interspersed with questions. However, we hypothesized that this experience could have been jarring or distracting for students (given that a lecture could not be viewed continuously). Similarly, that model would not allow students to answer questions at their own pace; rather, videos would be interrupted by questions at predetermined times, which could be a disruptive experience for students. Orthogonally, to ascertain the effectiveness of different teaching methods, we also wanted a framework for presenting distinct videos and assessments pertaining to the same topic to different students. As a result, we set out to create a generalized video player that could be embedded into any web page and assess students midvideo, which could ultimately help instructors improve students’ learning experience. SOLUTION To create a more interactive learning environment for students, we implemented CS50 Video, with support for desktop (and laptop) browsers as well as for mobile devices. With CS50 Video, instructors can present questions at predetermined times in a video that can be answered by students at their own pace without leaving the environment of the video player. Furthermore, CS50 Video gives students the ability to search through the transcript of any video and skip to the point at which a topic was discussed. In addition, CS50 Video supports captions and transcripts in multiple languages using the standardized SRT format and Google Translate’s API. Finally, CS50 Video allows students to view videos at variable playback speeds (both slower and faster), making videos more accessible for students who are less familiar with English (or unable to keep up with an instructor’s natural pace). Desktop CS50 Video’s desktop interface builds an interactive experience into a video player embeddable on any web page by allowing assessment questions and transcripts to be shown in the player itself. Using CS50 Video’s JavaScript API, instructors can associate a list of questions with a particular video. At a point in the video specified by the instructor, CS50 Video will notify students that a new question is available, and students can view the topics to which the question pertains. Students can choose to answer the question at any point in time, if the instructor permits, at which point the video player will flip around to display the selected question, as seen in Figure 1. CS50 Video offers built-in support for multiple-choice, true/false, numeric, and free-response (i.e., students’ answers are matched against a regular expression) questions. Instructors can also choose whether the correctness of answers is evaluated on the client (for simplicity) or on a server (for integrity). The utilization of server-side correctness checking also allows instructors to track students’ progress and view which questions are most difficult for students. In addition to integrating assessments, CS50 Video also allows students to browse and search over transcripts in multiple languages. CS50 Video utilizes Google’s Translate API to automatically translate transcripts into a number of languages if human-translated transcripts are unavailable. Figure 1: Video’s desktop player flips around to reveal assessments that can be answered by students at their own pace and supports transcripts in multiple languages. Not only do assessment questions have the potential to increase engagement levels among students, but responses may also provide insights to instructors. For example, students’ performance on assessment questions may reveal which course topics are potentially confusing, allowing instructors to give additional treatment to those topics before examinations or projects. Similarly, instructors can use analytics data describing what portions of a video were skipped over or repeated by students to improve the efficiency and clarity of lectures. Video’s potential for A/B-testing will also allow instructors to present different explanations or assessments of the same topic in order to determine quantitatively which methods are most effective. CS50 Video’s desktop interface is available as open-source software at http://github.com/cs50. Mobile We also wanted to provide students with a more compelling experience on mobile devices. In particular, smart phones and tablets, like the iPhone, iPad, and Android devices, offer the potential for an even more immersive experience, because of their full-screen displays and touch interfaces. In CS50 Video’s mobile interface, we strove to bring the classroom experience in an even greater extent to mobile devices. The mobile version of CS50 Video is a video player designed specifically for the classroom. As seen in Figure 2, students can view a video at variable playback speeds as well as view subtitles. In addition, students can search the transcripts, providing the ability to jump to various points in the lecture based on keyword searches, just as in CS50 Video’s desktop interface. In addition, CS50 Video’s mobile player affords students an immersive classroom experience. Students can toggle transparent “overlays” that display materials of the instructor’s choosing. For example, an instructor might choose to offer the lecture’s slides, notes, or source code for the students’ perusal. Alternatively, an overlay may consist of a second video feed, such as a recording of the instructor’s screen, that will be automatically synchronized with the video. This particular feature allows online learners to have the same flexibility as on-campus students; where an on-campus student can utilize a hard copy of the lecture slides or glance up at the screen to reference the slides, a student using CS50 Video’s mobile player can toggle the appropriate overlay. Overlays can be fully interactive, allowing students to scroll and pan through materials as they see fit. Students can also adjust the opacity of the overlays, depending on whether they would like to see more of the lecture content or the overlay content. Seeking to provide an even more faithful mobile classroom experience, we also enable students to open a full-screen web browser without leaving the (virtual) classroom. In this way, students can quickly search topics or terms that are foreign or confusing to them online. Providing students access to external resources and websites within the confines of the app itself eliminates the potential distraction of having to close the app, navigate to another app (e.g., to check mail), and reopen the app. CS50 Video’s mobile interface is also available as open-source software at http://github.com/cs50. RESULTS In Fall 2012, we deployed CS50 Video to 86,400 students online. Using CS50 Video’s analytics and server-side logging, we have been able to track students’ performance on assessment questions. Of the 86,400 students who have watched at least one video online, 24,000 (28%) have answered at least one in-video assessment question. After the first three months of the course, a total of 742,000 responses to assessment Figure 2: CS50 Video’s iPad player offers an immersive experience in which students can view overlays with supplementary material as well as search transcripts and the web. questions were recorded across 35 videos; 61.3% of these responses were correct answers. Although the semester is still in progress, we were disappointed with some of these early results, particularly the relatively low percentage (28%) of students who’ve engaged with CS50 Video’s interactive assessments. Vis-à-vis alternative players, it is possible we overcorrected with CS50 Video’s design, placing too much control in students’ own hands, whereby it is now too easy to overlook available questions. At the same time, we suspect students might not perceive sufficient value in answering questions, given that responses are recorded but not graded for credit. We intend to experiment further with alternative interfaces that might lend themselves to more active engagement. As seen in Figure 3, many correct answers were reached on students’ first attempts. However, many students continued to attempt questions until they reached a correct answer. Of the 520,000 total correct responses, 67.0% were reached on the first attempt, 80.3% in at most two attempts, and 84.5% in at most three attempts. FUTURE WORK Now that we have deployed CS50 Video to our online students, we plan to perform additional analysis on student data and introduce new features into the player. As soon as the semester comes to a close (in April 2013), we will compare question response rates to students’ overall performance in the course in order to assess the influence of in-video assessments on students’ master of material. We will also utilize the player in A/ B testing in order to compare to the player, which will allow us to compare the effectiveness of different questions on learning outcomes. We also intend to leverage students’ responses in order to improve the course’s curriculum. For instance, if a large percentage of students struggle with a topic, we can redress students’ confusion in a future lecture or section. Finally, we plan to develop new types of assessment questions. In particular, we will add support for coding problems, in which students will be able to write small programs whose correctness can be automatically assessed through server-side unit tests. In future iterations of CS50 Video’s mobile player, we will leverage the extra screen space afforded by tablets for extra content and functionality. In particular, we plan to integrate assessment questions into CS50 Video’s mobile interface as well and also allow students to take notes or bookmark important or confusing sections of the lecture content. Figure 3: Students answer a majority of questions correctly on the first attempt, but many students continue to respond to the question until they reach a correct answer. CONCLUSIONTraditional video players consisting only of basic playback controls to view course videos results in alargely passive experience for students. Not only are students unable to check their understanding of conceptsexplored in videos, but instructors lack data describing which videos are confusing or effective for students. Andso, we developed and deployed CS50 Video, a video player for desktop and mobile devices that allows studentsto answer a range of instructor-defined assessment questions at their own pace while providing detailed analyticsto instructors, to 86,400 students online. Students using CS50 Video’s mobile player can engage in a moreimmersive classroom experience by viewing lecture slides and notes concurrently with lecture videos. Using thedata obtained from CS50 Video, we plan to ascertain which topics were most confusing for students in order torefine the course’s curriculum and improve students’ learning experience. We also plan to compare students’performance on CS50 Video’s interactive questions and answers against final grades in order to evaluate thepredictive value of the embedded questions themselves. REFERENCES[1] Coursera. Coursera. http://www.coursera.org/[2] EdX. edX. http://www.edx.org/[3] J. Foertsch, G. Moses, J. Strikwerda, and M. Litzkow. Reversing the lecture/homework paradigm usingeteach web-based streaming video software. In 0-7803-7444-4/02/$17.00 (c) 2002 IEEE November 6 9, 2002,Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference T2F-26, pages 267–274, 2002.[4] Google, Inc. YouTube. http://www.youtube.com/[5] W. S. Harwood and M. M. McMahon. Effects of integrated video media on student achievement andattitudes in high school chemistry. Journal of Research in Science Teaching, 34(6):617–631, 1997.[6] T. Liu and J. Kender. Lecture videos for e-learning: current research and challenges. In Multimedia SoftwareEngineering, 2004. Proceedings. IEEE Sixth International Symposium on, pages 574 – 578, dec. 2004.[7] A. Mavlankar, P. Agrawal, D. Pang, S. Halawa, N.-M. Cheung, and B. Girod. An interactive region-of-interest video streaming system for online lecture viewing. In Packet Video Workshop (PV), 2010 18thInternational, pages 64 –71, dec. 2010.[8] The Khan Academy. Khan Academy. http://www.khanacademy.org/.[9] Udacity, Inc. Udacity | Free Online Courses. http://www.udacity.com/.[10] D. Zhang, L. Zhou, R. O. Briggs, and J. F. N. Jr. Instructional video in e-learning: Assessing the impact ofinteractive video on learning effectiveness. Information & Management, 43(1):15 – 27, 2006.